Mass storage device and method of operating the same to store parity data

ABSTRACT

A mass storage memory device is disclosed. The device includes a plurality of blades where two blades are used to store parity data corresponding to data stored in the other blades. The device also includes a controller configured to write data to the blades along stripes extending from the other blades to the two blades, where the parity data within a stripe is based on the data written to the other blades in the stripe, and wherein the parity data includes two or more types of parity data.

REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.61/793,591 filed Mar. 15, 2013, which is hereby incorporated in itsentirety and for all purposes.

FIELD OF THE INVENTION

The present invention relates, generally, to mass storage devices and,more particularly, to methods of storing and recovering redundant data,and systems implementing the methods.

BACKGROUND OF THE INVENTION

Mass storage systems are used to store large amounts of data. Importantparameters of such devices include speed and reliability. The systemspreferably operate without error for long uninterrupted periods of time.To accomplish this, the systems store system data in addition to userdata. The system data may be used to recover user data which is lostbecause of, for example, a power failure or a hardware failure.

Some systems use RAID (redundant array of independent disks) technology.RAID technology uses multiple memory components to form a single logicalmemory storage unit. The stored data is distributed among the memorycomponents, and includes the system data for data recovery. Dependingupon what level of RAID technology is used, the system may be able torecover from multiple errors. For example, RAID technology allows forrecovery from multiple errors. For example, RAID6 technology allows forrecovery from two errors.

SUMMARY OF THE INVENTION

One implementation is a mass storage device, including a plurality ofblades, where two of the blades are configured to store parity data andthe other blades are configured to store data written to the device. Thedevice also includes a controller configured to write data to the bladesalong stripes extending from the other blades to the two blades, and towrite parity data to the two blades, where the parity data within astripe is based on the data written to the other blades in the stripe,and where the parity data includes two types of parity data.

One inventive aspect is a mass storage device. The device includes aplurality of blades, where two of the blades are configured to storeparity data and the other blades are configured to store data written tothe device. The device also includes a controller configured to writedata to the blades along stripes extending from the other blades to thetwo blades, and to write parity data to the two blades. The parity datawithin a stripe is based on the data written to the other blades in thestripe, and where the parity data includes two types of parity data.

Another inventive aspect is a method of writing data to a mass storagedevice having a plurality of blades, where each blade includes aplurality of pages. The method includes receiving data to be stored,writing data corresponding to the received data along a stripe of themass storage device to a first subset of the blades, and calculatingfirst and second types of parity data, where the parity data iscalculated based on the written data. The method also includes writingthe first and second types of calculated parity data along the stripe toa second subset of the blades.

Another inventive aspect is a method of operating a mass storage devicehaving a plurality of blades. The method includes receiving anindication that one of the blades has failed, receiving an instructionto access data located on the failed blade, recovering data of thefailed blade, and executing the instruction using the recovered data.

Another inventive aspect is a method of operating a mass storage devicehaving a plurality of blades and a controller. The method includes thecontroller operating the device with all of the blades, the controllerreceiving an indication that one of the blades has failed, and thecontroller operating the device with the non-failing blades. The methodalso includes the controller receiving an indication that the failedblade has been replaced, and the controller operating the device withall of the blades.

Another inventive aspect is a method of operating a mass storage devicehaving a plurality of blades. The method includes determining an age ofthe device, and selecting an error recovery method based on the age ofthe device, where in the selected error recovery method is selected fromthe group consisting of ECC, RAID, and bad column mapping.

Another inventive aspect is a method of operating a mass storage devicehaving a plurality of blades. The method includes operating the devicewith all of the blades, receiving an indication that one of the bladesis at least partially unavailable, operating the device with theavailable blades, and recovering data of the unavailable blade based ondata stored in the available blades.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate implementations of inventiveconcepts and, together with the description, serve to explain variousadvantages and principles of the invention.

FIG. 1 is a block diagram illustrating a mass storage device.

FIG. 2 is a schematic diagram illustrating a single die.

FIG. 3 is a flowchart diagram illustrating an implementation of a methodof writing data to a mass storage device.

FIG. 4 is a flowchart diagram illustrating an implementation of a methodof recovering data in a mass storage device.

DETAILED DESCRIPTION OF THE INVENTION

Reference is made to implementations illustrated in the accompanyingdrawings. The same reference numbers are generally used throughout thedrawings and the following description to refer to the same or likeparts.

FIG. 1 is a block diagram illustrating a mass storage device 100. Insome implementations, the mass storage device 100 includes SLC (singlelevel cell) NAND flash memory technology. The device 100 mayadditionally or alternatively include one or more of MLC (multilevelcell), NOR, PCM, Spin-Torque, MRAM, Memsistors, or other technologies.

As shown, the mass storage device 100 includes multiple blades 110,where each of the blades 110 includes a controller 150 and multiplememory hardware devices 120. In some implementations, memory hardwaredevices 120 are located on both front and back surfaces of each of theblades 110. The mass storage device 100 may also include a systemcontroller (not shown) configured to cause the mass storage device 100to perform the operations and actions described herein.

As an example, the mass storage device 100 may include 24 blades 110,and each of the blades 110 may include 32 memory hardware devices 120.Each of the memory hardware devices 120 may include 64 GB of storagecapacity. In such an implementation, each of the blades 110 has 2 TB ofmemory storage capacity, and the mass storage device 100 has 48 TB ofmemory storage capacity. The mass storage device 100 also includes acontroller 150, which is configured to control the read, write, anderase operations of the mass storage device 100. In someimplementations, the number of blades 110, the number of memory hardwaredevices 120 on each blade 110, and/or the amount of storage capacityincluded in each of the memory hardware devices 120 may be different.

In some implementations, each of the memory hardware devices 120includes multiple die. For example, each of the memory hardware devices120 may include four die. FIG. 2 is a schematic diagram illustrating asingle die 200. As shown, the die 200 includes two planes 210, whereeach of the planes includes multiple blocks 220, and each of the blocks220 includes multiple pages 230. In addition, each of the pages 230includes multiple memory cell locations.

As an example, each of the pages 230 may include 128K bits (or memorycell locations). Furthermore, each of the blocks may include 256 pages230, and each of the planes 210 may include 2¹¹ blocks 220. Such a diehas a storage capacity of 16 GB. In some implementations, the number ofplanes 210, the number of blocks 220 in each plane 210, and/or thenumber of pages 230 in each block 220 may be different.

In some implementations, the planes 210 can be separately andsimultaneously written, read, and erased. For some memory technologies,each time data is written to or read from the die, an entire page iswritten or read. For some memory technologies, each time data is erasedan entire block of data is erased.

In some implementations, data written to the mass storage device 100 iswritten in stripes. A stripe includes one or more pages 230 from each ofmultiple blades 110. In some implementations, each stripe includes oneor more pages 230 from all of the blades 110. For example, a stripe mayinclude one page 230 from each plane 210 of one or more die 200 of eachmemory hardware device 120 on each of the blades 110.

In order to implement RAID technology in an n blade mass storage devicewhere stripes extend across all of the blades, data may be written ton−2 of the blades along one of the stripes, and parity data based on thedata written to the n−2 blades may be written along the stripe in thelast 2 blades. The parity data is written to the last 2 blades such thateach bit of each page of the last 2 blades corresponds with the paritydata of a set of corresponding bits of corresponding pages of the datawritten to the n−2 blades, where each set of corresponding bits includesone bit per blade.

In some implementations, one of the last 2 blades receives parity dataof a first type and the other of the last 2 blades receives parity dataof a second type. Various types of parity data may be used. For example,xor of the data written to the n−2 blades, and Reed Solomon parity dataor square of xor parity data may be used.

In some implementations, the parity data for a stripe is calculated aseach page or other portion of the stripe is written. Alternatively, theparity data for the data of the stripe in the n−2 blades may becalculated after the data is written to the n−2 blades.

In some implementations, the last 2 blades are not always the same 2blades. Instead, which two blades are used for parity informationchanges. For example, a first two blades may be assigned for use asparity data storage for a first stripe, and a second two blades may beassigned for use as a data storage for a second stripe. This may beadvantageous at least because the parity information is not read duringnormal operation, and distributing the parity data among all of theblades balances the read load across the blades.

In some implementations, the controller 150 on each of the blades 110 isconfigured to perform an error correction function. Each controller 150is configured to detect, and attempt to correct data errors which haveoccurred on the blade 110 associated therewith. If an error has occurredwhich cannot be corrected by a controller 150, the mass storage device100 may correct the error using the parity data stored in the last 2blades 110 of the device 100. If a single error has occurred, the paritydata of one of the 2 types of parity data, for example, the xor paritydata, may be used to correct the error. If two errors have occurred, theparity data of both of the 2 types of parity data may be used to correctthe errors.

In some systems, in order to implement RAID technology in an n blademass storage device where stripes extend across all of the blades, datamay be written to n−m of the blades along one of the stripes, and paritydata based on the data written to the n−m blades may be written alongthe stripe in the last m blades, where m is three or more, such as RAID7or RAID8 technology. The parity data is written to the last m bladessuch that each bit of each page of the last m blades corresponds withthe parity data of a set of corresponding bits of corresponding pages ofthe data written to the n−m blades, where each set of corresponding bitsincludes one bit per blade.

In some implementations, each of the last m blades receives parity dataof a different type. In some implementations, one or more of the last mblades receives parity data which is the same type as the parity datareceived by one or more others of the last m blades. Various types ofparity data may be used. For example, xor of the data written to the n−mblades, and Reed Solomon parity data or square of xor parity data may beused.

In some implementations, the parity data for a stripe is calculated aseach page or other portion of the stripe is written. Alternatively, theparity data for the data of the stripe in the n−m blades may becalculated after the data is written to the n−m blades.

In some implementations, the last m blades are not always the same mblades. Instead, which blades are used for parity information changes.For example, a first m blades may be assigned for use as parity datastorage for a first stripe, and a second m blades may be assigned foruse as a data storage for a second stripe. This may be advantageous atleast because the parity information is not read during normaloperation, and distributing the parity data among all of the bladesbalances the read load across the blades.

In some implementations, if an error has occurred which cannot becorrected by a controller 150, the mass storage device 100 may correctthe error using the parity data stored in the last m blades 110 of thedevice 100. If a single error has occurred, the parity data of one ofthe m types of parity data, for example, the xor parity data, may beused to correct the error. Likewise, if two or more errors haveoccurred, the parity data of two or more types of parity data may beused to correct the errors.

In some circumstances, errors can be caused by the failure of a blade,causing the data stored on the blade to be lost. The blade failure mayinclude a failure of the entire blade, or a failure of one or moreportions of the blade. For example, a blade failure may include afailure of any of a plane, a block, a page, a die, a memory hardwaredevice, a controller, and any other portion of a blade which renders theblade partially or wholly inoperative. The blade failure mayadditionally or alternatively include a circumstance in which any of ablade, a plane, a block, a page, a die, a memory hardware device, acontroller, and any other portion of a blade which renders the bladepartially or wholly unavailable. For example, when the system performsan operation, such as a reset, an erase, or programming operation, theblade or a portion of the blade may be occupied by the operation andtherefore unavailable or inaccessible for, for example, a data readoperation. In such circumstances, the effect of, for example,“replacing” the blade is achieved by the blade becoming available afterthe occupying operation has completed.

Conventionally, mass storage systems experiencing the failure of a blademust use data throughout the entire memory system to recover the lostdata. Accordingly, to recover the lost data, all of the data stored inthe memory is rebuilt. Such recovery is extremely time-consuming, andmust be performed prior to the conventional mass storage system beingoperational following the failure.

In the mass storage device 100, because the parity information stored inthe last 2 blades and is stored by stripe, data lost because of thefailure of a blade can be recovered stripe by stripe. For example, usingknown techniques, which vary according to the type of parity used, dataunavailable because of the failure of a blade may recovered byregenerating or calculating the unavailable data based on the paritybits and the data stored in the other blades of each stripe. Forexample, for each stripe, there may be 2 parity bits. To regenerate thedata in a particular stripe of an unavailable blade, the unavailabledata is calculated based on the data in the particular stripe of theremaining available blades and the 2 parity bits for the particularstripe.

Because the process of recovering the data of each stripe issufficiently fast, the data lost because of the blade failure can berecovered as needed. For example, if a read operation is to beperformed, and the location of the data to be read includes a page whichhas not been recovered, the system may then, in response to the readoperation, recover the data of the lost page. In some instances, somepages may be erased before the data stored therein is needed. For suchpages, the data is not needed and may not be recovered.

Further reducing the impact of a blade failure, in some implementations,the allocation of memory capacity to applications being served may bethinly provisioned. Thin provisioning allows for memory capacity whichis been allocated, but not used, to be shared by multiple applications.For example, each of the applications using mass storage device 100 maybe allocated an amount of storage capacity corresponding to need andtype of application, where the total amount of capacity allocated to theapplications is greater than the actual physical capacity of the massstorage device 100. For example, mass storage device 100 may have atotal physical capacity of 44 TB, but the total of capacity allocated tothe applications may be 100 TB.

In such thinly provisioned systems, the memory storage capacity for eachapplication is virtually allocated thereto. Accordingly, physicalsections of the mass storage device are not assigned to specificapplications a priori, but instead, are assigned as they are used. Withthis allocation scheme, the data within the mass storage device for eachapplication tends to become segmented and unorganized. To minimize thiseffect, part of the normal operation of a thinly provisioned device maybe to move data from location to location in order to have the datastored in a more optimized configuration.

Because the normal operation of a thinly provisioned device includesrearranging and reorganizing data, the impact of a blade failure may beminimal. If such a blade failure occurs, the system may note that thefailed blade is unavailable for writing and erasing, and may continue tooperate normally. In some embodiments, if data is to be read from thefailed blade, the lost data from the failed blade is regenerated usingthe parity bits as discussed above, and is rewritten elsewhere in thememory to a blade which is operational.

Likewise, once the failed blade is replaced with an operational blade,the system may note that the new blade is available for reading,writing, and erasing. Because the mass storage device 100 is configuredto continue to operate despite having a failed blade and to continue tooperate despite having a newly replaced blade, utilization time of themass storage device 100 is maximized and performance is optimized.

FIG. 3 is a flowchart diagram illustrating an implementation of a methodof writing data to a mass storage device, such as the mass storagedevice 100. The mass storage device implements RAID technology torecover from errors. The mass storage device may be, for example, usedby multiple applications simultaneously for storing and retrieving datarelated to the operation of the applications.

In step 310, data is received which is to be written to storage device100. The data may be received from one of the applications incommunication with the mass storage device 100. The data may be modifiedin preparation for storage. For example, the data may be rearranged orsegmented so as to be written along a stripe of the mass storage device100. In some embodiments, preparation for storage includes compressingthe data.

In step 320, the data is written along a stripe extending acrossmultiple blades configured for storage of application data. As the datais written, at step 330, a determination is made as to whether thepreviously written data was written to the last page of the last bladeconfigured for storage of application data. If the previously writtendata was not written to the last page of the last blade configured forstorage of application data, additional data is written in step 320. Ifthe previously written data was written to the last blade configured forstorage of application data, parity data is calculated and written insteps 340 and 350.

In step 340, parity data of a first type is calculated as discussedabove. The parity data of the first type is stored along the same stripeas the data written in step 330 in one of two blades configured forstorage of parity data. The first type of parity data may, for example,be xor data based on the data written in step 330.

In step 350, parity data of a second type is calculated as discussedabove. The parity data of the second type is stored along the samestripe as the data written in step 330 in the other of the two bladesconfigured for storage of parity data. The second type of parity datamay, for example, be squared xor or Reed Solomon data based on the datawritten in step 330.

In systems which use additional parity bits, additional parity bits ofthe same or additional parity types are calculated and stored along thesame stripe as the data written in step 330 to additional bladesconfigured for storage of the additional parity bits.

FIG. 4 is a flowchart diagram illustrating an implementation of a method400 of recovering data in a mass storage device, such as the massstorage device 100, where the data in the mass storage device has beenpreviously written using, for example, the method illustrated in FIG. 3.The mass storage device implements RAID technology for recovery ofmultiple errors. The mass storage device may be, for example, used bymultiple applications simultaneously for storing and retrieving datarelated to the operation of the applications.

In step 410, a failure of one of the blades of the mass storage device100 is detected. The failure, for example, may be caused by a loss ofpower to the failed blade. Once the blade has failed, the datapreviously written to the blade is inaccessible. The blade is noted asbeing unavailable for writing and erasing.

In step 420, an instruction is received which requires accessing datawhich was stored on the failed blade. The instruction, for example, maybe instructions to read data which was stored on the failed blade.

In step 430, data which was stored on the failed blade is recovered. Torecover the data, the data may be regenerated using parity data asdiscussed above. The regenerated data may then be written to one of theoperational blades using, for example, one or more aspects of themethods discussed above.

In step 440, the instruction is executed using the recovered data. Forexample, if the instruction is to read data which was stored on thefailed blade, data corresponding to the data on the failed blade is readfrom the operational blade to which the recovered data was written.

In some implementations, aspects of the methods and systems describedabove can be used to implement additionally or alternatively other errorrecovery schemes. For example, in some implementations, ECC, RAID, andbad column mapping may be used, for example, by statically ordynamically changing schemes to optimize capacity and recoverycapabilities for example, as a device ages. For example, a recoveryscheme may be selected based at least in part on an indication of an ageof a device.

While various embodiments of present invention have been described, itwill be apparent to those of skill in the art that many more embodimentsand implementations are possible that are within the scope of thisinvention. Accordingly, the present invention is not to be restrictedexcept in light of the attached claims and their equivalents.

What is claimed is:
 1. A mass storage device, comprising: a plurality ofblades, wherein two of the blades are configured to store parity dataand the other blades are configured to store data written to the device;and a controller configured to write data to the blades along stripesextending from the other blades to the two blades, and to write paritydata to the two blades, wherein the parity data within a stripe is basedon the data written to the other blades in the stripe, and wherein theparity data includes two types of parity data.
 2. The mass storagedevice of claim 1, wherein the controller is configured to allocatestorage of the blades to applications, wherein the allocation is thinlyprovisioned.
 3. The mass storage device of claim 1, wherein thecontroller is configured to recover data of a failed blade in responseto an operation to access the data of the failed blade.
 4. The massstorage device of claim 1, wherein three or more of the blades areconfigured to store parity data and the other blades are configured tostore data written to the device, and wherein the controller isconfigured to write data to the blades along stripes extending from theother blades to the three or more blades, and to write parity data tothe three or more blades, wherein the parity data within a particularstripe is based on the data written to the other blades in theparticular stripe, and wherein the parity data includes three or moretypes of parity data.
 5. A method of writing data to a mass storagedevice comprising a plurality of blades, each blade comprising aplurality of pages, the method comprising: receiving data to be stored;writing data corresponding to the received data along a stripe of themass storage device to a first subset of the blades; calculating firstand second types of parity data, wherein the parity data is calculatedbased on the written data; and writing the first and second types ofcalculated parity data along the stripe to a second subset of theblades.
 6. The method of claim 5, wherein the first and second types ofparity data are calculated in response to data of the received databeing written to a last page of a last blade of the first subset.
 7. Themethod of claim 5, wherein the second subset of blades comprises twoblades.
 8. The method of claim 5, further comprising: calculating athird type of parity data based on the written data; and writing thethird type of calculated parity data along the stripe to the secondsubset of blades, wherein the second subset of blades comprises three ormore blades.
 9. A method of operating a mass storage device comprising aplurality of blades, the method comprising: receiving an indication thatone of the blades has failed; receiving an instruction to access datalocated on the failed blade; recovering data of the failed blade; andexecuting the instruction using the recovered data.
 10. The method ofclaim 9, wherein recovering the data of the failed blade comprises:retrieving data from the plurality of blades, the retrieved datacomprising a plurality of data bits and a plurality of parity bits; andcalculating the data of the failed blade based on the retrieved data andparity bits.
 11. The method of claim 10, wherein the retrieved data bitsand parity bits are retrieved from blades which have not failed, and areretrieved from stripes which include the data of the failed blade.
 12. Amethod of operating a mass storage device comprising a plurality ofblades and a controller, the method comprising: the controller operatingthe device with all of the blades; the controller receiving anindication that one of the blades has failed; the controller operatingthe device with the non-failing blades; the controller receiving anindication that the failed blade has been replaced; and the controlleroperating the device with all of the blades.
 13. The method of claim 12,further comprising the controller allocating storage of the blades toapplications, wherein the allocation is thinly provisioned.
 14. Themethod of claim 12, further comprising the controller recovering data ofthe failed blade in response to an operation to access the data of thefailed blade.
 15. A method of operating a mass storage device comprisinga plurality of blades, the method comprising: determining an age of thedevice; and selecting an error recovery method based on the age of thedevice, where in the selected error recovery method is selected from thegroup consisting of ECC, RAID, and bad column mapping.
 16. The method ofclaim 15, further comprising: operating the device with all of theblades; receiving an indication that one of the blades has failed;receiving an instruction to access data located on the failed blade;recovering data of the failed blade based on the selected error recoverymethod; and executing the instruction using the recovered data.
 17. Amethod of operating a mass storage device comprising a plurality ofblades, the method comprising: operating the device with all of theblades; receiving an indication that one of the blades is at leastpartially unavailable; operating the device with the available blades;and recovering data of the unavailable blade based on data stored in theavailable blades.
 18. The method of claim 17, further comprising: priorto receiving the indication, executing an operation which causes data ofthe at least partially unavailable blade to become unavailable.
 19. Themethod of claim 18, wherein recovering the data of the unavailable bladecomprises: retrieving data from the plurality of blades, the retrieveddata comprising a plurality of data bits and a plurality of parity bits;and calculating the data of the unavailable blade based on the retrieveddata and parity bits.
 20. The method of claim 19, wherein the retrieveddata bits and parity bits are retrieved from blades which are available,and are retrieved from stripes which include the data of the unavailableblade.